With the advent of the digital camera, and now with the highly competitive market in which many companies such as CANON™, SONY™ and NIKON™ have a mass of models for sale to the general public at very affordable prices, a very large number of people that own a computer also own a digital camera. Largely due to this, a blooming area in web-based utilities is the online storage of personal photo albums.
Existing applications such as YAHOO!™ Photos and FLICKR™ allow users to upload their images onto a remote database, thereby saving space by not having to store them locally. However, these online databases must not only store the photographs, they must also be able to display them, and provide some methods of searching and sorting through them.
Present methods of image searching include content-based scans such as color pattern recognition and histogram statistics. Although research is currently being conducted into this field, these types of searches are not very accurate, and will not be perfected for many years to come.
Furthermore, the image display organization of many of these existing web applications is confusing, and does not clearly denote an order in the images, namely there is no clear definition of which image is more relevant than the others.
Numerous methods for searching, sorting, and displaying images have been proposed over the years [1-18]. (References to background documents indicated by Arabic numbers in square brackets refer to one or more of the references cited in the “List of References” below.) The methods and systems of searching images can be generally divided into content-based and tag-based categories. Content-based image searching/retrieval systems have the benefit of being mostly automatic (requiring little or no user tagging) with the ability to search virtually any image based on what it contains [1-9]. Yet, they suffer from the problem of requiring extensive computations for the search, making a real-time implementation on a large-scale image database expensive. Furthermore, content-based systems often result in inaccurate search results due to the misrecognition of the contents of the image, thereby resulting in frustrated system users and a perception of unreliability. While a variety of modifications to classic content-based image search have been proposed over the years, such as user feedback and sub-image analysis [10], the level of reliability of these systems for general applications have not been increased to the point of wide-spread user acceptance.
An alternative and more successful method, such as the one implemented on websites such as FLICKR™, is to manually tag or describe the contents of an image, and then to search these images by searching the tags or descriptions of the images [11,12]. Tag-based methods have the advantage of being reasonably reliable (as long as the people tagging can be trusted) while requiring very simple computations (as compared to content-based approaches). However, tagging images requires significant time and effort to be spent by the user especially for large image databases. Furthermore, tag-based approaches are unable to identify the specific location of the object being searched within an image; all they can do is to identify whether the object exists in the image or not.
Many methods for searching, sorting, or tagging images have been patented. One example is a patent that provides a method and apparatus for assigning keywords to objects found within images, as well as video and audio files [4]. These keywords can be assigned to file names or URLs of images. Images can also be described by the text in the link, including headings and titles, captions, or ALT text HTML tags used to annotate images. Images can thus be searched through these descriptions or “tags”. This method of searching takes into account the fact that not all captions found within the URL are useful or relevant to specific images, and thus it only tags images using the text found closest to them. It also determines whether any words in the text match the name assigned to the image, thus text that matches is given higher priority as a tag. This method can only be successful if the images are carefully named or described within their URL. However, it cannot locate specific objects within images, or images themselves if they have not been appropriately described by the user.
A lot of related patents use image processing to search for images. An “automatic person meta-data labeler” offers a method of labeling objects within images, and does so by detecting their distinctive colors [18]. It is useful for the prioritization and sorting of images as well as for identifying a designated person that appears in a group of images. It can be used to automatically label images by detecting particular patters and colors, such as a piece of jewelry that a person is wearing in all images, or their skin tone. As useful as this is for labeling pictures within certain albums or groups of images taken of one scene from different angles, it may not be useful for pictures taken on different days or of various sceneries. It also may require a lot of computational ability, and may take a long time to process and label large groups of images.
Another similar patent provides a method for the searching of images according to queries containing visual attributes of images, including color, texture, shape, and size [2]. These queries also contain textual tags attached to images. This is useful in retrieving images, by entering their visual or textual attributes, and searching the query. It is especially effective in searching for images by queries based on pixel information of contained images.
This type of image search is also carried out by a patent that provides an indexing method for accelerating a content-based image search [3]. It can identify the most similar images by analyzing their color, texture, shape, or content, and generating and storing index values.
Color is a common attribute used to define images, and most methods use it to search for images. One patent provides this type of method of image searching by using a multilevel image grid data structure of different hierarchical levels with respect to one color in an image [15]. These grid color levels are compared and matched, grouping similar images.
Another method allows an image search by using a histogram, comprised of bins [17]. For example, a histogram of each image in a database is created using its color information, in order of importance. If a particular color appears in a lot of images, it is given a low priority, and placed at the end of the chart. Similarly, a less common color would be given high priority and positioned at the beginning of a chart. Therefore, images that contain less frequent colors will be more distinguishable and would thus be effective bins to use.
Other methods of searching that have been implemented to break images down into segments. One example is a patent that involves searching for and localizing an object within an image, by breaking it down into a number of geometric sub-models [5]. Geometric descriptions of objects can then be specified in some coordinate system, which enables object localization for a broad range of imaging conditions. This is an efficient method both in memory and computational time.
Another method segments images into regions to which weights are assigned, and calculates the similarity between images by comparison of set weights [13]. The weights are calculated by comparing feature similarities for each region in a plurality of images. It is thus a useful method for searching for similar images.
Another patented system can be used to search for images by storing codes in a database [14]. For example, it stores facial codes which define facial features in each image into a database, linking them to corresponding images, instead of storing whole images themselves. This allows for image searching, while reducing the amount of necessary storage, making this a fast and efficient method.
These methods can all be extended to further applications, such as video and audio searches. One example of a previously implemented apparatus is one developed for the application of a surveillance camera system. This apparatus searches for a desired image through a set of images recorded by the camera, using the difference in luminance information between the reference and the other recorded images [16].
What is needed therefore is novel methods and systems for searching, sorting, and/or displaying a set of images stored in a database that overcomes the limitations of the prior art.